{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4-4AutoGraph的机制原理\n",
    "\n",
    "有三种计算图的构建方式：静态计算图，动态计算图，以及Autograph。\n",
    "\n",
    "TensorFlow 2.0主要使用的是动态计算图和Autograph。\n",
    "\n",
    "* 动态计算图易于调试，编码效率较高，但执行效率偏低。\n",
    "\n",
    "* 静态计算图执行效率很高，但较难调试。\n",
    "\n",
    "* Autograph机制可以将动态图转换成静态计算图，兼收执行效率和编码效率之利。\n",
    "\n",
    "当然Autograph机制能够转换的代码并不是没有任何约束的，有一些编码规范需要遵循，否则可能会转换失败或者不符合预期。\n",
    "\n",
    "我们会介绍Autograph的编码规范和Autograph转换成静态图的原理。\n",
    "\n",
    "并介绍使用tf.Module来更好地构建Autograph。\n",
    "\n",
    "上篇我们介绍了Autograph的编码规范，本篇我们介绍Autograph的机制原理。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一、Autograph的机制原理\n",
    "\n",
    "**当我们使用@tf.function装饰一个函数的时候，后面到底发生了什么呢？**\n",
    "\n",
    "例如我们写下如下代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np \n",
    "\n",
    "@tf.function(autograph=True)\n",
    "def myadd(a,b):\n",
    "    for i in tf.range(3):\n",
    "        tf.print(i)\n",
    "    c = a + b\n",
    "    print(\"tracing\")\n",
    "    return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tracing\n",
      "0\n",
      "1\n",
      "2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'helloworld'>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 后面什么都没有发生。仅仅是在Python堆栈中记录了这样一个函数的签名。\n",
    "myadd(tf.constant(\"hello\"),tf.constant(\"world\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发生了2件事情，\n",
    "\n",
    "第一件事情是创建计算图。\n",
    "\n",
    "即创建一个静态计算图，跟踪执行一遍函数体中的Python代码，确定各个变量的Tensor类型，并根据执行顺序将算子添加到计算图中。\n",
    "在这个过程中，如果开启了autograph=True(默认开启)，会将Python控制流转换成TensorFlow图内控制流。\n",
    "主要是将if语句转换成 tf.cond算子表达，将while和for循环语句转换成tf.while_loop算子表达，并在必要的时候添加 tf.control_dependencies 指定执行顺序依赖关系。\n",
    "\n",
    "相当于在 tensorflow1.0执行了类似下面的语句：\n",
    "\n",
    "```python\n",
    "g = tf.Graph()\n",
    "with g.as_default():\n",
    "    a = tf.placeholder(shape=[], dtype=tf.string)\n",
    "    b = tf.placeholder(shape=[], dtype=tf.string)\n",
    "    cond = lambda i: i<tf.constant(3)\n",
    "    def body(i):\n",
    "        tf.print(i)\n",
    "        return(i+1)\n",
    "    loop = tf.while_loop(cond, body, loop_vars=[0])\n",
    "    loop\n",
    "    with tf.control_dependencies(loop):\n",
    "        c = tf.strings.join([a, b])\n",
    "    print(\"tracing\")\n",
    "```\n",
    "\n",
    "第二件事情是执行计算图。\n",
    "\n",
    "相当于在 tensorflow1.0中执行了下面的语句：\n",
    "\n",
    "```python\n",
    "with tf.Session(graph=g) as sess:\n",
    "    sess.run(c, feed_dict={a:tf.constant(\"hello\"), b:tf.constant(\"world\")})\n",
    "```\n",
    "\n",
    "因此我们先看到的是第一个步骤的结果：即Python调用标准输出流打印\"tracing\"语句。\n",
    "\n",
    "然后看到第二个步骤的结果：TensorFlow调用标准输出流打印1,2,3。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**当我们再次用相同的输入参数类型调用这个被@tf.function装饰的函数时，后面到底发生了什么？**\n",
    "\n",
    "例如我们写下如下代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'goodmorning'>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myadd(tf.constant(\"good\"),tf.constant(\"morning\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只会发生一件事情，那就是上面步骤的第二步，执行计算图。\n",
    "\n",
    "所以这一次我们没有看到打印\"tracing\"的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**当我们再次用不同的的输入参数类型调用这个被@tf.function装饰的函数时，后面到底发生了什么？**\n",
    "\n",
    "例如我们写下如下代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tracing\n",
      "0\n",
      "1\n",
      "2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=3>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myadd(tf.constant(1),tf.constant(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于输入参数的类型已经发生变化，已经创建的计算图不能够再次使用。\n",
    "\n",
    "需要重新做2件事情：创建新的计算图、执行计算图。\n",
    "\n",
    "所以我们又会先看到的是第一个步骤的结果：即Python调用标准输出流打印\"tracing\"语句。\n",
    "\n",
    "然后再看到第二个步骤的结果：TensorFlow调用标准输出流打印1,2,3。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**需要注意的是，如果调用被@tf.function装饰的函数时输入的参数不是Tensor类型，则每次都会重新创建计算图。**\n",
    "\n",
    "例如我们写下如下代码。两次都会重新创建计算图。因此，一般建议调用@tf.function时应传入Tensor类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tracing\n",
      "0\n",
      "1\n",
      "2\n",
      "tracing\n",
      "0\n",
      "1\n",
      "2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'goodmorning'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myadd(\"hello\",\"world\")\n",
    "myadd(\"good\",\"morning\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 二、重新理解Autograph的编码规范\n",
    "\n",
    "了解了以上Autograph的机制原理，我们也就能够理解Autograph编码规范的3条建议了。\n",
    "\n",
    "1、被@tf.function修饰的函数应尽量使用TensorFlow中的函数而不是Python中的其他函数。例如使用tf.print而不是print.\n",
    "\n",
    "解释：Python中的函数仅仅会在跟踪执行函数以创建静态图的阶段使用，普通Python函数是无法嵌入到静态计算图中的，所以在计算图构建好之后再次调用时，这些Python函数并没有被计算，而TensorFlow中的函数则可以嵌入到计算图中。使用普通的Python函数会导致\n",
    "被@tf.function修饰前【eager执行】和被@tf.function修饰后【静态图执行】的输出不一致。\n",
    "\n",
    "2、避免在@tf.function修饰的函数内部定义tf.Variable. \n",
    "\n",
    "解释：如果函数内部定义了tf.Variable,那么在【eager执行】时，这种创建tf.Variable的行为在每次函数调用时候都会发生。但是在【静态图执行】时，这种创建tf.Variable的行为只会发生在第一步跟踪Python代码逻辑创建计算图时，这会导致被@tf.function修饰前【eager执行】和被@tf.function修饰后【静态图执行】的输出不一致。实际上，TensorFlow在这种情况下一般会报错。\n",
    "\n",
    "3、被@tf.function修饰的函数不可修改该函数外部的Python列表或字典等数据结构变量。\n",
    "\n",
    "解释：静态计算图是被编译成C++代码在TensorFlow内核中执行的。Python中的列表和字典等数据结构变量是无法嵌入到计算图中，它们仅仅能够在创建计算图时被读取，在执行计算图时是无法修改Python中的列表或字典这样的数据结构变量的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
